AD  689751 


AFHRL-TR-69-101 


April  1969 


Field  Test  of  the  Weighted  Airman  Promotion  System: 
Phase  I.  Analysis  of  the  Promotion  Board  Component 
in  the  Weighted  Factors  System 


By 

Janos  B.  Koplyay 


This  document  has  been  approved  for  public  release  and  sa'e. 
its  distribution  is  unlimited 


PERSONNEL  RESEARCH  DIVISION  p 

AIR  FORCE  HUMAN  RESOURCES  LABORATORY  ^ 


n 


AIR  FORCE  SYSTEMS  COMMAND 
Lackland  Air  Force  8s>e,  Texas 


-i'  ■- 


:rnfA 


il!{  JUL  l--1969  ;i|l 

I  '  V •  u 


Rep»o.^«-^  br  :>vc 

CLEARINGHO 
f.iera*  A 

Inlorr.'Ai'On  Sc^'r 


USE 

Tochn.':a! 
Va  22J5I 


£0 


AFRHL-TR-69-101 


April  1969 


FIELD  TEST  OF  THE  WEIGHTED  AIRMAN  PROMOTION  SYSTEM: 
PHASE  I.  ANALYSIS  OF  THE  PROMOTION  BOARD  COMPONENT 
IN  THE  WEIGHTED  FACTORS  SYSTEM 


By 

JmtmB.  Kortray 


Thi»  document  hat  been  approved  for  public  release  and  sale; 
its  distribution  a  unlimited. 


PERSONNEL  RESEARCH  DIVISION 
AIR  FORCE  HUMAN  RESOURCES  LABORATORY 
AIR  FORCE  SYSTEMS  COMMAND 
Lackland  Air  Force  Base,  Texas 


FOREWORD 
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F.L.  McLanathan,  Lt  Col,  USAF 
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ABSTRACT 


For  the  purposes  of  field  testing  the  proposed  Weighted  Airman  Promotion  System, 
the  Alaskan  Air  Command  supplied  data  on  2,835  airmen  eligible  for  promotion  to  grades 
E4  through  E-7  in  the  FY  69-B  promotion  cycle.  Data  included  scores  on  the  Specialty 
Knowledge  Test  and  the  Promotion  Fitness  Examination;  points  credited  for  time  in 
grade,  time  in  service,  decorations,  and  Airman  Performance  Report  mean  overall 
evaluation;  and  ID  information.  Cases  for  which  test  scores  were  unavailable  were 
eliminated;  the  final  sample  consisted  of  2,290  airmen,  81  percent  of  the  original 
eiigibles. 

Average  promotion  board  scores  were  computed  by  dividing  each  airman’s  raw 
board  score  by  the  number  of  members  on  the  promotion  board  and  multiplying  the 
quotient  by  10.  Weighted  factors  scores  were  computed  by  adding  the  factor  scores.  Two 
such  total  composite  scores  were  obtained:  a  weighted  factors  score  excluding  the  average 
board  score  and  a  weighted  factors  score  including  the  average  board  score.  The  analyses 
concentrated  on  the  comparisons  of  these  two  total  composite  scores.  The  hypothesis 
tested  was  that  inclusion  of  the  board  score  component  in  the  weighted  factors  composite 
score  does  not  have  an  effect  on  the  ranking  of  the  airmen.  Ranks  on  the  composite  total 
scores  with  and  without  the  board  score  were  analyzed  for  strength  of  relationship. 
Average  board  scores  were  analyzed  for  differences  in  scoring  across  Air  Force  Bases  and 
by  different  size  board  panels. 

The  following  results  were  obtained:  («)  There  were  significant  differences  in 
average  board  score  means  between  Qmendorf  and  Eielson  AFBs  for  grade  E-3  personnel. 
(b)  There  were  significant  differences  in  average  board  score  means  between  3-member 
and  5-member  board  panels,  (c)  There  was  a  very  high,  near  perfect  relationship  between 
weighted  factors  composites  excluding  and  including  the  board  score  component,  (d) 
Rankings  of  individuals  on  the  weighted  factors  composite  were  essentially  unaffected  by 
inclusion  of  the  promotion  board  score. 

Since  inclusion  of  a  board  score  component  had  a  negligible  effect  on  the  ranking 
by  composite  total  score,  and  since  means  of  board  scores  differed  between  bases  and 
between  boards  with  different  panel  sizes,  it  was  concluded  that  including  a  board  score 
factor  merely  inserted  a  non-visible  component  which  would  obscure  explanations  for 
non-promotion. 
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FIELD  TEST  OF  THE  WEICHTED  AIRMAN  PROMOTION  SYSTEM: 
PHASE  I.  ANALYSIS  OF  THE  PROMOTION  BOARD  COMPONENT 
IN  THE  WEIGHTED  FACTORS  SYSTEM 


I.  BACKGROUND 

As  an  outgrowth  of  the  requirements  for  efficient,  equitable  promotion  procedures  which  permit 
consideration  of  an  optimum  number  of  variables  in  the  selection  of  airmen  for  promotion,  a  selection 
model  has  been  designed  and  proposed  for  implementation.  Part  of  the  initial  phase  of  this  work,  the  field 
testing  of  the  Weighted  Airman  Promotion  System,  is  the  subject  of  this  report. 

Under  existing  procedures  in  the  Air  Force,  airman  promotions  arc  carried  out  by  means  of 
promotion  boards.  These  boards,  consisting  of  three  or  five  panel  members,  are  convened  for  each 
promotion  cycle  to  review  a  given  number  of  folders  of  airmen  who  are  eligible  for  promotion.  Each  board 
member  assigns  points  for  each  airman  (not  less  than  1  and  not  more  than  10  points)  to  reflect  his 
judgment  about  the  merits  of  the  performance  documented  in  the  airman’s  folder.  Promotion  board  scores, 
i.c..  the  sum  of  the  scores  for  each  airman  given  by  these  panel  members,  are  reported  to  the  major 
commands.  The  airmen  are  ranked  from  liigh  to  low  on  their  board  scores,  ties  in  rankings  are  broken  by 
crediting  points  for  time  in  grade,  time  in  service.  Airman  Performance  Report  rating,  and  decorations. 
Then  selections  for  promotion  arc  made  according  to  quotas  assigned  for  the  different  Air  Force 
Specialties.  Going  down  from  high  to  low  on  the  promotion  board  score  ranking,  the  quota  for  each 
specialty  is  filled. 

During  February  and  March  1968,  research  was  conducted  at  the  Personnel  Research  Division  to 
develop  a  model  for  a  new  airman  promotion  system  that  could  substitute  for  the  present  promotion 
system.  Hopefully,  the  new  system  would  provide  more  visibility  to  each  airman  in  terms  of  numeric  values 
on  selected  variables,  which  would  in  turn  provide  the  airmen  with  an  indication  of  how  to  increase  his 
probability  for  promotion,  or  would  give  him  evidence  and  information  about  the  rezsons  he  was  not 
promoted. 

The  brief  description  of  this  initial  research  which  follows  provides  a  background  for  the  present 
study.  A  group  consisting  of  1 5  officers  (7  Colonels,  2  Lt  Colonels,  and  6  Majors)  and  16  noncommissioned 
officers  (3  Senior  Master  Sergeants  and  13  Chief  Master  Sergeants)  served  as  teams  to  provide  judgments  of 
over  two-thousand  airmen  who  were  eligible  for  promotion  from  grade  E-5  to  £-6.  These  selected  judges 
were  extremely  well  qualified,  highly  experienced  personnel  who  were  among  the  most  competent  persons 
in  the  Air  Force  to  determine  the  relative  merits  of  airmen  eligible  for  promotion.  The  task  of  the  Personnel 
Research  Division  was  to  capture  the  policy  of  these  judges  in  ranking  the  individual  airmen  or,  in  other 
words,  to  build  a  mathematical  model  identifying  and  reproducing  as  closely  as  possible  the  relative  weights 
of  the  variables  used  by  the  judges  in  ranking  the  airmen  for  promotion.  A  model,  derived  from  the 
selection  policy  of  the  largest,  most  consistent,  and  homogeneous  group  of  judges,  was  recommended  for 
adoption.  Tins  model  with  its  weighting  of  selected  variables  gives  the  visibility,  equity,  and  consistency 
over  time  desired  in  a  system  used  to  promote  enlisted  personnel. 

The  merits  and  desirability  of  thr  new  weighted  factors  promotion  system  were  discussed  at  length 
during  the  USAF  Airman  Promotion  Conference  held  in  April  1968.  The  genera!  consensus  was  that 
selection  for  promotion  could  be  based  cither  completely  or  to  a  great  extent  on  the  mathematical  model 
consisting  of  the  major  selection  factors,  with  little  or  no  requirement  for  inclusion  of  a  promotion  board 
component  as  is  currently  used.  The  alternate  solution  of  providing  promotion  boards  with  weights  and 
factors  to  consider  appeared  to  introduce  excessive  cost  without  enhancing  the  degree  of  visibility  of  the 
individuals  under  consideration.  After  more  intensive  discussion,  the  proposed  weighted  factors  system 
developed  by  the  Personnel  Research  Division,  with  factors  and  weights  as  proposed  by  the  Division  and 
modified  by  the  panel  of  judges,  was  recommended  for  implementation  at  the  earliest  feasible  date. 
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The  weighted  factors  promotion  system  was  approved  by  the  Secretary  of  the  Air  Force  on 
3  July  1968  and  was  subjected  to  a  field  test  in  the  Alaskan  Air  Command.  The  purpose  of  the 
field  test  was  to  obtain  tangible  information  with  regard  to  the  feasibility  of  the  new  system 
itself,  to  determine  the  relative  contribution  of  a  promotion  board  score  to  the  weighted  factors 
composite  score,  and  to  validate  the  Promotion  Fitness  Examination.  The  field  test  can  thus  be 
summarized  under  three  major  phases: 

1.  Analysis  of  the  effects  of  the  promotion  board  score  on  predictive  efficiency  of  the 
weighted  factors  composite  score. 

2.  Validation  of  the  weighted  Airman  Promotion  System. 

3.  Validate  i  of  the  Promotion  Fitness  Examination. 

This  report  is  a  summary  of  the  first  phase  dealing  with  the  effects  of  the  promotion  board  score  on 
the  proposed  weighted  factors  system.  The  second  and  third  phases,  i.e.,  validation  of  the  weighted  factors 
system  itself  and  validation  of  the  Promotion  Fitness  Examination,  are  reported  separately  because  these 
phases  attempt  to  answer  entirely  different  kinds  of  questions. 


II.  EXPLANATION  OF  TOE  WEIGHTED  FACTORS  SYSTEM 

The  components  derived  from  the  initial  analysis  of  the  selection  policy  of  the  panel  of  judges 
included  scores  for  the  Specialty  Knowledge  Test,  the  Promotion  Fitness  Examination,  time  in  grade,  time 
in  service,  decorations,  and  the  mean  overall  evaluation  from  Airman  Performance  Reports,  with  a  possible 
additional  component  -  a  promotion  board  score.  Appendix  1  shows  the  factors  and  the  maximum  points 
for  each,  with  an  explanation  of  their  use  and  computation  for  grades  E-4  through  E-7.  Also  indicated  are 
two  alternative  scoring  procedures,  one  with  the  inclusion  of  a  promotion  board  evaluation  and  the  other 
with  no  promotion  board  component. 

It  should  be  emphasized  that  the  points  indicated  in  Appendix  I  are  maximum  obtainable  points. 
There  is,  of  course,  a  very  small  probability  that  the  maximum  total  points  can  be  obtained  by  any  one 
airman.  In  order  to  obtain  the  maximum  number  of  points,  an  airman  would  have  to  obtain  perfect  scores 
on  the  factors  involving  tests  (190  points),  have  a  minimum  of  20  years  in  service  (40  points),  have  10  years 
in  grade  (60  points),  and  have  been  awarded  decorations  for  the  maximum  number  of  points  (25  points: 
Medal  of  Honor,  15  points;  Air  Force  Cross,  9  points;  and  one  other  decoration  or  seme  equivalent 
recognition  for  heroism,  1  point).  The  small  probability  of  occurrence  of  a  maximum  total  score  limits  the 
interpretation  of  these  points  in  terms  of  percentage  of  contribution  to  the  total  score  as  shown  in  the 
appendix.  Thus,  no  attempt  should  be  made  to  assert  that  the  score  on  the  Specialty  Knowledge  Test,  for 
instance,  “contributes”  17  percent  to  the  total  score,  or  that  the  contribution  of  decorations  is  5  percent. 
In  fact,  the  percentage  of  contribution  of  an  individual  factor  depends  entirely  on  the  individual  scores  on 
these  factors  in  each  airman’s  case.  The  percentages  indicated  m  the  appendix  are  valid  only  if  the  values  for 
all  factors  are  maximum.  More  explicitly,  the  percentages  shown  are  not  predetermined  but,  rather,  are 
derived  from  the  values  in  this  particular  example,  showing  in  this  case  the  relative  contribution  of  each 
factor  to  the  total  score. 

Another  point  that  should  be  kept  in  mind  is  that  any  weighted  system  is  in  a  sense  an  arbitrary 
judgment  of  the  group  of  people  assigning  the  weights.  In  this  instance,  the  weights  derived  and  proposed 
were  based  on  the  captured  policy  of  a  team  of  Air  Force  managers.  There  is  no  guarantee  that  the  policy 
of  this  group,  or  the  weights  arrived  at  by  capturing  their  policy,  would  necessarily  be  strongly  related  to 
the  judgment  or  policy  of  any  one  of  the  existing  promotion  boards.  In  fact,  it  is  entirely  possible  that 
policies  and  weights  used  by  different  promotion  boards  would  have  differential  values  if  analyzed 
mathematically.  The  aim  of  validation,  however,  will  be  to  find  an  indicator  expressing  the  degree  of 
agreement  between  the  lists  of  incumbents  selected  by  a  promotion  board  and  those  selected  by  the 
weighted  factors  system.  Validation  of  the  system  is  discussed  in  a  separate  report. 

In  the  present  discussion,  the  contribution  of  the  promotion  board  score  to  the  weighted  factors 
composite  is  considered,  as  well  as  the  possibility  and  effects  of  eliminating  this  factor  altogether.  The 
nature  of  this  discussion  forces  two  basic  assumptions:  First,  that  the  weighted  factors  promotion  system  is 
accepted  for  implementation;  and  second,  that  the  weights  used  are  accepted  as  proposed.  The  only  topic 
addressed  by  this  report  is  the  question  of  retention  or  elimination  of  the  promotion  board  component. 
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ID.  PROCEDURE 


Data  Collection 

Data  for  the  study  were  collected  on  personnel  in  the  Alaskan  Air  Command.  The  data  were  compiled 
during  the  period  S  August  1968  to  1  October  1968  at  Elmendorf  and  Eielson  Air  Force  Bases  for  airmen 
in  pay  grades  E-3  through  E-6  who  were  eligible  for  promotion  during  the  FY  1969-B  promotion  cycle. 
f  These  cases  did  not  include  airmen  at  remote  sites  for  whom  no  attempt  was  made  to  obtain  Promotion 

Fitness  Examination  scores.  Data  included  Specialty  Knowledge  Test  scores,  Airman  Performance  Report 
scores,  Time-in*Gradc  scores,  Time-in-Service  scores,  Decorations  scores,  and  Promotion  Board  scores.  The 
'»  latter  were  given  in  raw  score  form;  the  maximum  possible  promotion  board  score  was  30  for  3-member 

panels  and  50  for  5-member  panels.  Elmendorf  Air  Force  Base  had  both  3-member  and  5-member  panels, 
while  Eielson  used  5-member  panels  only.  The  score  card  work-deck  format  is  shown  as  Appendix  II.  In 
addition  to  the  factors  described,  the  data  cards  included  other  dates  and  information,  together  with 
computed  total  composite  score.  This  total  score,  however,  did  not  include  the  Promotion  Fitness 
Examination  score.  Further,  the  inclusion  of  a  promotion  board  score  with  differing  upper  limits  made  it 
necessary  to  recompute  a  weighted  factors  composite  score  in  which  the  board  score  component  was 
adjusted  for  differences  in  panel  size.  Of  the  2,835  cases  for  whom  data  were  received,  some  were  excluded 
either  because  no  Specialty  Knowledge  Test  scores  were  available  or  no  Promotion  Fitness  Examination 
scores  were  available.  Two  other  exclusions  were  caused  by  out-of-range  data.  Table  1  summarizes  the 
sample  used  in  the  study. 


Table  l.  Study  Sample  by  Pay  Grade 
(W  -  2,290) 


Cat** 

e-a 

e-4 

e-s 

e-c 

Airman  Eligible  for  Promotion 

605 

1,130 

619 

481 

Cases  Excluded 

50 

280 

105 

1 10 

Total  for  Pay  Grade 

555 

850 

514 

371 

Promotion  Fitness  Examination  Score 

An  experimental  form  of  the  Promotion  Fitness  Examination  was  administered  to  personnel  in  the 
Alaskan  Air  Command.  Answer  sheets  from  a  total  of  2,578  examinees  were  scored  at  the  Personnel 
Research  Division  and  were  added  to  the  information  already  received  on  those  examinees  from  the 
Alaskan  Air  Command. 

Time -in -Grade  and  Time-in-Service  Scons 

To  assure  maximum  accuracy,  Time-in-Gradc  and  Time-in-Service  scores  were  recomputed  using  Date 
of  Rank  (DOR)  and  Total  Active  Federal  Military  Service  Date  (TAFMSD)  and  the  date  1  January  1969  for 
airmen  in  grade  E-3  and  I  May  1969  for  airmen  in  grades  E-4  through  E-6.  Results  were  distributed  against 
the  Time-in-Grade  and  Time-in-Service  scores  supplied  by  the  Alaskan  Air  Command.  There  was  only  one 
case  for  which  the  given  Time-in-Service  score  differed  from  the  computed  one;  this  discrepancy,  of  course, 
was  corrected. 

Average  Promotion  Board  Score 

Before  total  composite  weighted  scores  could  be  computed,  a  decision  was  necessary  regarding  use  of 
the  promotion  board  scores  in  their  raw  score  form  since  the  promotion  panels  were  of  two  different  sizes. 
Means  and  standard  deviations  were  computed  by  grade,  by  Air  Force  base,  and  by  number  of  panel 
members.  These  computations  were  used  to  determine  whether  scores  would  have  to  be  standardized  or 
whether  some  kind  of  average  board  score  could  be  used  with  an  adjustment  for  the  difference  in  panel  size. 
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Tabic  2  summarizes  the  means,  standard  deviations,  and  number  of  eases  in  each  category  for  the 
promotion  board  raw  scores.  U  is  apparent  front  the  table  that  variances  were  similar  enough  to  permit 
computation  of  an  Average  Board  Score  for  each  airman.  This  was  done  for  each  airman  by  dividing  his 
score  by  the  number  of  panel  members  and  multiplying  this  quotient  by  10  to  obtain  a  range  between  1 
and  100  as  required  and  shown  in  Appendix  1  (Board  Evaluation  maximum  points  =  100). 

Table  3  summarizes  the  means  and  standard  deviations  of  the  computed  Average  Board  Scores  by 
grade,  by  Air  Force  base,  and  by  number  of  panel  members.  A  fairly  sizable  difference  is  indicated  between 
the  means  of  airmen  in  grade  H-3  with  5-membcr  panels  (mean  =  79.46)  and  3-:nembcr  panels  (mean  = 
63.67).  This  dtffercncc  (79.46  -  63.67  =  15.79)  is  statistically  significant  beyond  the  .0001  level  of 
confidence.  One  might  conclude,  then,  that  a  difference  between  the  two  means  of  these  groups  could 
happen  less  than  once  out  of  10,000  times  by  chance  alone.  There  appears  to  be  a  strong  interaction 
between  the  size  of  the  board  panel  and  the  mean  score.  A  comparison  between  the  means  of  grade  F.-3 
airmen  at  the  two  bases  for  groups  having  5-member  panels  reflects  a  statistically  significant  difference 
beyond  the  .01  level  of  confidence.  In  other  words,  the  difference  between  these  means  (79.46  for 
Elmendorf  and  75.18  for  Eielson)  could  happen  onlv  less  than  once  out  of  100  times  by  chance  alone. 
Although  no  other  significant  differences  are  apparent  from  Tables  2  and  3,  it  is  safe  to  conclude  that 
promotion  board  scores  m  certain  instances  show  significant  differences  when  compared  between  different 
locations  or  even  within  the  same  location  when  the  panel  size  is  different.  This  conclusion  cannot  be 
generalized  from  the  available  data  since  this  study  deals  with  only  two  locations;  however,  there  is  a  strong 
likelihood  that  one  would  find  similar  differences  dealing  with  other  promotion  boards. 


Table  2.  Promotion  Board  Raw  Score  Means,  Standard  Deviations, 
and  Number  of  Cases  by  Grade,  AFB,  and  Panel  Size 


Raw  Scores 


Sndt 

Board 
Mam  ban 

Elmendorf 

Eielson 

Total 

N 

Mean 

so 

N 

Mean 

so 

N 

Mean 

SD 

E-3 

5 

38 

39.73 

5.17 

129 

37.59 

3.25 

167 

37.95 

4.23 

3 

388 

19.10 

3.49 

- 

- 

- 

388 

19.10 

3.49 

E-4 

5 

612 

36.87 

4.65 

238 

36.53 

4.53 

850 

36.77 

4.62 

E-5 

5 

406 

34.76 

4.00 

108 

34.75 

3.29 

514 

34.77 

3.70 

E-6 

5 

301 

37.94 

4.22 

70 

38.74 

3.66 

371 

38.08 

4.13 

Total 

3 

388 

. 

388 

19.10 

3.49 

5 

1,357 

545 

1.902 

36.59 

4.43 

j 


Table  3.  Average  Board  Score  Means,  Standard  Deviations,  and 
Number  of  Cases  by  Grade,  AFB,  and  Panel  Size 


l 


I 

I 
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Average  Board  Scores  Times  10 


Elmendorf 


Grade 

Board 

Members 

N 

Mean 

SD 

E-3 

5 

38 

79.46 

10.34 

3 

388 

63.67 

11.63 

E-4 

5 

612 

73.74 

9.30 

E-5 

5 

406 

69.52 

8.00 

E-6 

5 

301 

75.88 

8.44 

Total 

3 

388 

5 

1,357 

Eielson 

Total 

N 

Mean 

so 

N 

Mean 

so 

129 

75.18 

6.50 

167 

75.90 

8.46 

- 

- 

- 

388 

63.67 

11.63 

238 

73.06 

9.06 

850 

73.54 

9.24 

108 

69.50 

6.58 

514 

69.54 

7.40 

70 

77.48 

7.32 

371 

76.16 

8.26 

- 

388 

63.67 

11.63 

545 

1.902 

73.18 

8.86 

545 


Weighted  Factors  Score 

The  individual  variables  were  distributed  by  grade.  Weighted  factors  composite  scores  were  computed 
two  ways:  First,  the  weighted  factors  composite  score  excluding  the  average  promotion  board  score 
(Weighted  Factors  Score  Without  Average  Board  Score)  and  second,  the  weighted  factors  composite  score 
including  the  average  promotion  board  score  (Weighted  Factors  Score  With  Average  Board  Score). 

Distribution  of  the  Average  Board  Scores  revealed,  as  expected,  a  high  incidence  of  tics.  Appendix  II! 
shows  the  number  of  ties  and  the  number  of  actual  promotions  at  the  promotion  quota  cutoff  points  by 
grade.  For  instance,  for  grade  E-4,  there  were  21 2  promotions  (quota  cutoff  point).  Out  of  the  total  sample 
of  airmen  in  grade  E-4  (850),  there  were  66  ties  in  Average  Board  Score  for  rank  212.  Twenty-nine  of 
these  66  ties  were  actually  promoted;  37  of  the  ties  were  not  promoted. 

The  high  incidence  of  ties  reflects  the  limited  discrimination  power  of  board  scores  when  grades  arc 
analyzed  without  further  breakdown  by  Control  Air  Force  Specialty  Codes  (CAFSC).  However,  ven 
though  the  number  of  ties  within  a  particular  CAFSC  may  be  smaller,  the  probability  of  ties  in  board  scores 
is  still  much  larger  than  the  probability  of  ties  in  weighted  factors  total  scores.  The  implication  is  that  there 
is  a  better  discrimination  using  the  Weighted  Factors  Score  With  Average  Board  Score  than  there  is  using 
the  Average  Board  Score  alone.  One  could  argue  that  in  the  case  of  a  tie,  the  board  “takes  another  look”  at 
the  individuals’  records  in  order  to  break  the  tie.  However,  one  could  also  argue  that  if  this  “other  look” 
differentiates  the  individuals,  then  either  the  “first  look”  was  erroneous  (that  is,  there  should  not  have  been 
a  tie),  or  this  “second  look”  is  a  factor  which  is  invisible  to  the  airman  and  is  not  detectable  or  measurable 
by  any  particular  mathematical  model. 


IV.  EFFECTS  OF  THE  AVERAGE  BOARD  SCORE  ON  THE  WEIGHTED  FACTORS  SCORE 

Results  Obtained  for  Total  Sample  with  a  Maximum  Board  Score  Component  of  100 

The  basic  question  of  this  phase  of  the  investigation  is:  What  effect,  if  any,  does  inclusion  of  the 
promotion  board  component  in  the  weighted  factors  system  have  on  the  ranking  of  individuals? 

To  answer  this  question,  each  airman  within  each  grade  was  ranked  on  three  variables:  (a)  Weighted 
Factors  Score  Without  Average  Board  Score,  ( b )  Weighted  Factors  Score  With  Average  Board  Score,  and  (c) 
Average  Board  Score.  Analyses  were  performed  at  different  hypothetical  promotion  percentage  quotas  of 
20,  30,  40,  and  50  percent.  That  is,  airmen  were  ranked  by  Weighted  Factors  Score  Without  Average  Board 
Score,  and  the  first  20  percent  of  the  particular  grade  (1 14  for  E-3,  for  example)  were  analyzed  in  terms  of 
matches  between  these  ranks  and  ranks  on  the  Weighted  Factors  Score  With  Average  Board  Score.  These 
percentages  were  computed  as  +1.5  percent  to  accommodate  for  ties.  For  example,  for  grade  F.-3,  20 
percent  of  555  is  111.  Because  of  ties,  the  quota  number  114  was  chosen  instead.  Of  these  114  airmen, 
there  were  102  whose  ranks  on  the  Weighted  Factors  Score  Without  Average  Board  Score  and  on  the  Weighted 
Factors  Score  With  Average  Board  Score  were  both  better  than  1 14.  Trie  ratio  of  102  to  1 14  (89.5  percent) 
represents  the  overlap  between  the  two  ranks.  This  ratio  indicates  the  effects  of  adding  the  board  score 
component  to  the  weighted  factors  score.  That  is,  89.5  percent  of  the  individuals  who  would  have  been 
promoted  using  the  Weighted  Factors  Score  Without  Average  Board  Score  would  have  also  been  promoted 
by  the  Weighted  Factors  Score  With  Average  Board  Score.  The  rank  overlap  and  overlap  percent  (or  index) 
is  defined  in  this  study  as  the  number  of  the  corresponding  percentage  of  individuals  who  could  be 
promoted  by  both  promotion  systems  when  the  two  systems  are  compared,  or  the  number  and  percentage 
of  individuals  who  would  actually  have  been  promoted  by  a  system  when  this  system  is  compared  with 
actual  promotions. 

Table  4  summarizes  these  rank  overlaps  by  grade.  The  table  also  includes  the  overlap  at  the  actual 
promotion  quota  cutoff  point  (478  for  grade  E-3.  for  example).  It  can  be  seen  from  the  table  that  the 
lowest  overlap  index  is  87  percent,  but  that  the  majority  of  these  percentages  3re  at  the  90  percent  level  or 
higher.  This  implies  that  the  ranking  of  individuals  in  terms  of  the  weighted  factors  Score  Without  Average 
Board  Score  dunged  very  little  when  ranking  was  in  term:  of  the  Weighted  Factors  Score  With  Average 
Board  Score. 
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Table  4.  Overlap  Between  Ranks  cn  Weighted  Factors  Scotes  With  and 
Without  Avenge  Board  Scon 


Hypo  tnat  leal 
Promotion 

Quota 

Promotion 
Quota  N 

NumMr 
Rank*  on 
Both  Scorn 
Evcaodtnf 
Quota  Cutoff 

Parca.it 

Airaomant 

Batwoan 

Rankt 

Promotion 
Quota  N 

Numbar 
Ranks  on 
Roth  Scorns 
Exceeding 
Quota  Cutoff 

Parcant 

Afraamant 

Batwaan 

Ranks 

Grade  E-3 

Grade  E-4 

20% 

114 

102 

89.5 

170 

153 

90.0 

30% 

166 

156 

94.0 

255 

230 

90.2 

40% 

222 

205 

923 

341 

323 

94.7 

50% 

269 

249 

92.6 

422 

400 

94.8 

Actual 

Promotions 

478 

467 

97.7 

212 

192 

90.6 

Grade  E-5 

Grade  E-6 

20% 

105 

100 

95.2 

77 

69 

89.6 

30% 

154 

147 

95.5 

110 

98 

89.1 

*0% 

207 

201 

97.1 

149 

139 

933 

50% 

257 

251 

97.7 

189 

177 

94.7 

Actual 

Promotions 

89 

81 

91.0 

46 

40 

87.0 

Table  5  summarizes  the  interconelations  between  the  Weighted  Factors  Score  Without  Average  Board 
Score  and  the  Weighted  Factors  Score  With  Average  Board  Score.  From  the  table,  it  is  clear  that  there  is  a 
strong  relationship  between  the  two  variables.  In  fact,  the  correlations  are  near  perfect.  Again,  it  is  clear 
that  adding  the  board  score  component  to  the  weighted  factors  score  has  very  little,  if  any,  effect  upon  the 
ranking  of  the  individuals  or  upon  their  total  weighted  factors  score.  In  other  words,  adding  the  beard  score 
has  the  effect  of  adding  a  near  constant  to  every  score,  thus  increasing  the  total  score  but  changing  the 
ranking  very  little.  Table  6  summarizes  the  interconelations  between  rankings  on  the  Weighted  Factors 
Score  Without  Average  Board  Score  and  the  Weighted  Factors  Score  With  Average  Board  Score. 


Tables.  Interconelations  Between 
Weighted  Factors  Scores  With  and 
Without  Average  Board  Score 


Qrada 

Intareo  mutton 

Batwaan 

Variants 

E-3 

.9741 

E-4 

.9814 

E-5 

.9857 

E-6 

.9826 

Table  6.  Interconelations  Between 
Ranks  on  Weighted  Factors  Scores 
With  and  Without  Avenge  Board  Score 

Cradt 

Intarcorratatlan 

Batwaan 

VarUMas 

E-3 

.9733 

E-4 

.9801 

E-5 

.9871 

E-6 

.9871 

Results  Obtained  for  Total  Sample  with  a  Maximum  Board  Score  Component  of  200 

A  maximum  of  100  points  for  the  proposed  board  score  component  for  the  weighted  factors  system 
was  specified  as  the  value  to  be  investigated  in  this  field  test.  The  assignment  of  this  maximum  value  was 
made  independently  from  the  assignment  of  values  for  the  other  factors  in  the  construction  of  the 
weighting  system.  Since  a  board  score  component  was  not  a  factor  in  the  policy  capturing  exercises  which 
yielded  the  weights  for  the  other  factors,  the  assignment  of  a  100-point  maximum  was,  in  a  sense,  a  rather 
arbitrary  decision.  Before  the  results  of  this  field  test  became  available,  it  was  impossible  to  predict  with 
any  assurance  whether  the  board  score  component  would  significantly  affect  the  ranking  of  incumbents  for 
promotion  as  determined  by  a  weighted  factors  system.  Results  of  the  preceding  analysis  indicated  that  a 
100-point  maximum  board  score  component  introduces  virtually  no  change  in  the  ranking  of  individuals 
based  on  a  weighted  factors  score  which  includes  no  board  score  component. 

In  an  effort  to  determine  whether  a  larger  board  score  component  would  produce  significant  changes 
in  the  ranking  of  individuals,  the  size  of  the  board  score  component  was  doubled,  and  the  relationships 
were  studied  between  the  Weighted  Factors  Score  Without  Average  Board  Score  and  a  score  with  the  board 
score  component  increased,  the  Weighted  Factors  Score  With  Accentuated  Average  Board  Score.  The 
maximum  board  score  for  this  trial  system  was  200  points.  Ranks  on  the  Weighted  Factors  Score  Without 
Average  Board  Score  were  compared  with  ranks  on  the  Weighted  Factors  Score  With  Accentuated  Average 
Board  Score,  These  comparisons,  together  with  comparisons  of  ranks  obtained  when  the  100-point 
maximum  board  score  was  used,  are  presented  in  Table  7. 

The  values  in  Table  7  imply  that  doubling  the  board  score  component  does  not,  in  general,  decrease 
the  overlap  between  Weighted  Factors  Scores  With  or  Without  Average  Board  Score.  For  example,  96.2 
percent  of  grade  E-4  personnel  who  would  have  been  promoted  by  the  weighted  factors  system  excluding 
the  board  score  component  would  also  have  been  promoted  by  the  weighted  factors  system  using  the 
200-point  maximum  board  score  component.  In  the  same  grade,  this  percentage  was  90.1  percent  when  the 
100-point  maximum  board  score  componert  was  used.  Doubling  the  size  of  the  board  score  component 
does  not  appear  to  decrease  the  number  of  matches  (overlap  ranks),  or  the  number  of  individuals  who 
would  be  promoted  by  both  systems  (Weighted  Factors  Scores  With  and  Without  Average  Board  Score).  It 
seems  apparent,  then,  that  even  with  its  value  doubled,  the  inclusion  of  a  board  score  component  in  the 
weighted  factors  composite  system  fails  to  influence  the  ranking  of  airmen. 


Table  7.  Overlap  Between  Ranks  on  Weighted  Factors  Scores  Without  Average  Board  Score 
vs.  Weighted  Factors  Scores  with  Average  Board  Score  (Using  100-Point  Maximum) 
and  With  Accentuated  Avenge  Board  Score  (Using  200-Point  Maximum) 


WejfhtM  radon  Seem 
Compand 

rromdlon 

Quota 

Cutoff 

ttumkar 
Ranks  on 
Botti  Scorn 
ExsooOlnf 
Quota  Cutoff 

rorcont 
Afroomant 
•  ataman 
Ranks 

Promotion 

Quota 

Cutoff 

Numkor 
Ranks  on 
lot*  Scorn 
Exc  mains 
Quota  Cutoff 

Percent 

A  j  ■  —S  m  1 1  st 

Batsman 

Ranks 

Without  Board  Score  vs. 

Giade  E-3 

Grade  E-4 

478 

429 

89.8 

212 

204 

96.2 

With  Accentuated  Board  Score 

Without  Board  Score  vr. 

With  Board  Score 

478 

467 

97.7 

212 

197 

90.1 

Grade  E-5 

Grade  E-6 

Without  Board  Score  vs. 

With  Accentuated  Board  Score 

89 

84 

94.4 

46 

46 

100.0 

Without  Board  Score  vs. 

With  Board  Score 

89 

81 

91.0 

46 

40 

87.0 
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Results  Obtained  for  Selected  CAFSCs 

The  sample  from  the  Alaskan  Air  Command  (N  =  2,290)  did  not  lend  itself  to  complete  analysis  by 
CAFSC.  Most  specialties  had  small  frequencies  of  eligible  airmen.  It  was  possible,  however,  to  select  a  few 
CAFSCs  with  enough  airmen  who  were  eligible  for  promotion  to  conduct  a  meaningful  mathematical 
analysis.  Table  8  shows  the  number  of  eligible  airmen  by  grade  and  by  CAFSC.  Some  of  the  frequencies 
were  rather  small  due  to  the  attempt  to  analyze  the  same  CAFSCs  across  the  four  grades. 

Essentially  the  same  type  of  analysis  was  performed  on  these  selected  CAFSC  groups  as  was  carried 
out  in  the  case  of  the  total  grades.  For  each  CAFSC  group,  the  Weighted  Factors  Scores  Without  Average 
Board  Score  were  correlated  with  their  Weighted  Factors  Scores  With  Average  Board  Score.  These 
correlations,  also  shown  in  Tabie  3  by  grade  and  by  the  selected  CAFSCs,  serve  as  indices  of  the  effect  of 
adding  the  board  score  component  to  the  weighted  factors  score.  As  expected,  the  analysis  of  the  weighted 
factors  scores  with  and  without  the  board  score  showed  an  even  more  marked  relationship  when  broken 
down  into  CAFSCs.  The  near  perfect  correlations  ranged  from  .9726  to  .9941.  The  addition  of  the  board 
score  component  seems  to  have  little  or  no  effect  on  the  relative  ranking  of  the  individuals  on  the  weighted 
factors  score. 


Table  X.  Intercorrditions  Between  Weighted  Factors  Scores  With  and  Without  Average 
Board  Score  by  Selected  CAFSCs  and  by  Grade 


431 XX 

631 XX 

645 XX 

647XX 

702XX 

811XX 

N 

r 

N 

r 

N 

r 

N 

r 

N 

r 

N 

r 

E-3 

49 

.9768 

23 

.9836 

41 

.9726 

32 

.9806 

25 

.9789 

108 

.9770 

E-4 

55 

.9815 

74 

.9926 

57 

.9869 

35 

.9854 

50 

.9870 

35 

.9914 

Er5 

34 

.9879 

42 

.9838 

44 

.9851 

10 

.9941 

42 

.9675 

25 

.9843 

E-6 

31 

.9872 

17 

.9927 

25 

.9870 

11 

.9846 

33 

.9759 

13 

.9746 

V.  SUMMARY  AND  CONCLUSIONS 

A  fseld  test  of  the  weighted  factors  promotion  system  was  conducted  using  data  furnished  by  the 
Alaskan  Air  Command.  The  sample  included  2,290  airmen  in  pay  grades  E-3  through  E-6  who  were  eligible 
for  promotion.  The  analysis  of  this  report  concentrated  on  the  contribution  of  the  promotion  board 
component  to  the  weighted  factors  system  and  its  effects  on  the  relative  ranking  of  the  individuals. 

Promotion  board  scores  were  converted  to  a  scale  with  100  as  the  upper  limit.  Weighted  factors  scores 
were  computed  with  and  without  the  average  board  scores.  Factors  of  the  weighted  factors  system  were 
distributed  and  analyzed.  The  airmen  in  the  study  were  ranked  on  both  total  scores,  i.e„  total  weighted 
factors  scores  excluding  the  average  board  score  and  total  weighted  factors  scores  including  the  average 
board  score.  Overlaps  in  ranks  on  these  two  total  scores  were  computed  for  various  promotion  percentage 
quotas  and  for  the  actual  number  of  promoted  individuals  within  each  grade.  The  alternative  total  scores 
and  ranks  on  these  scores  were  also  analyzed  in  terms  of  correlations.  Separate  and  additional  analyses  were 
performed  for  selected  CAFSCs  where  there  were  sufficient  numbers  of  airmen  eligible  for  promotion  to 
permit  meaningful  analyses. 

The  following  results  were  found  from  the  analyses  performed: 

1.  For  airmen  in  grade  E-3,  promotion  board  scores  diffeied  significantly  for  3-member  and 
5-member  panels  at  Elmendorf  Air  Force  Base. 

2.  Mean  board  scores  at  Elmendorf  and  Eielson  Air  Force  Bases  differed  significantly  for  grade  E-3 
airmen,  suggesting  that  board  scores  may  in  part  be  dependent  on  base  of  assignment. 
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3.  inclusion  of  the  average  promotion  board  score  in  the  weighted  factors  score  had  little  or  no  effect 
on  the  relative  ranking  of  the  individuals  on  the  total  weighted  factors  score.  This  was  true  for  each  grade 
and  each  selected  CAFSC  analyzed. 

4.  Giving  more  weight  to  the  board  score  (i.e.,  doubling  its  value)  did  not  change  the  relative  ranking 
of  individuals,  inclusion  of  this  hypothetical  “heavy”  board  score  left  rankings  of  individuals  on  the 
composite  score  practically  unaffected. 

The  following  conclusions  are  based  on  the  two  fundamental  assumptions  stated  earlier,  i.e.,  that  the 
weighted  factors  promotion  system  is  accepted  as  the  new  system  for  promotion  and  that  the  weights 
assigned  to  the  factors  are  accepted  as  valid.  In  this  light,  there  is  conclusive  evidence  that  addition  of  a 
promotion  board  score  to  the  weighted  factors  score  is  unnecessary.  Analyses  of  this  study  dearly  show 
that  addition  of  the  board  score  to  the  composite  score  changes  very  little,  if  any,  the  ranking  of  the 
individuals  on  that  composite  score,  in  addition,  airmen  cannot  interpret  board  scores  since  the 
components  of  board  scores  are  not  visible  to  them.  Thus,  under  the  assumptions  of  this  study,  the 
inclusion  of  a  board  score  component  appears  to  be  unnecessary,  as  well  as  inconsistent  with  the  v^ibility 
objective  of  the  proposed  weighted  factors  promotion  sy.*em. 


APPENDIX  l:  AIRMAN  PROMOTION  SELECTION  FACTORS  AND  POINTS 

Grades  E-4  through  E-7 


Computation  With  Board  Score 

Computation  Without  Board  Score 

Maximum 

Maximum 

Sal  action  Factor 

Points 

Parcantaje 

Points 

Parctntaaa 

Specialty  Knowledge  Test  (SKT)  Score 

95 

17 

95 

21 

Promotion  Fitness  Examination  (PFE)  Score 

95 

17 

95 

21 

Timc-In-Service  (TIS)  Score 

40 

7 

40 

9 

Time-ln-Grade  (TIG)  Score 

60 

11 

60 

13 

Decoration  Score 

25 

5 

25 

6 

Airman  Performance  Reports  (APR) 

135 

25 

135 

30 

Board  Evaluation 

100 

18 

550 

100 

450 

100 

Explanation  of  Factors 

The  SKT  and  PFE  will  be  administered  annually. 

Points  for  the  SKT  and  PFE  scores  will  be  actual  percentile  scores  obtained  in  these  tests  (in  5-point 
increments). 

Time -in-Service  will  be  computed  by  multiplying  TAFMS  by  2.  Less  than  6  months  will  count  as  1  point; 
over  6  months  will  count  a  full  year,  2  points.  A  cutoff  score  of  40  points,  for  20  years  TAFMS,  has  been 
established. 

Time-in-Gradc  will  be  computed  at  the  rate  of  V:  point  per  month  up  to  a  maximum  of  120  months.  60 
points;  15  days  or  less  will  be  dropped,  16  or  more  will  count  as  a  full  month. 

Decorations  will  be  assigned  points  according  to  their  order  of  precedence.  The  maximum  number  of  points 
attainable  is  25.  Decorations  will  count  for  promotion  regardless  of  the  military  service  in  which  they  were 
earned. 

The  Airman  Performance  Report  score  is  obtained  by  multiplying  the  overall  evaluation  mean  by  1 5.  The 
mean  is  based  on  reports  for  a  5-year  period  prior  to  the  eligibility  date,  not  to  exceed  ten  reports. 

The  Board  Score  will  be  based  on  a  review  by  the  board  that  concentrates  on  those  items  not  previously 
weighted;  e.g.,  education  level  and  efforts  to  improve  self  in  terms  of  formal  education,  technical 
knowledge,  etc.  Reduced  selection  folder  will  consist  of  Category  A  favorable  communications,  APR  word 
picture,  and  pages  2  and  4  of  the  Air  Force  Form  7. 


APPF.SDIX  II.  SCORE  CARD  WORK-DECK  FORMAT 


DATA 
Not  Used 

Air  Force  Service  Number  (AFSN) 

Number  of  Beard  Members 
Name 

Date  of  Rank  (DOR) 

Total  Active  Federal  Military  Service  Datc(TAFMSD) 
Control  Air  Force  Specialty  Code  (CAFSC) 

Board  Score 

Airrran  Performance  Report  (APR)  Score 
Decoration  Score 

Specialty  Knowledge  Test  (SKT)  Score 

Time-in-Gradc  (TIG)  Score 

Timc-in-Scrvice  (TIS)  Score 

Not  Used 

Total  Score 

Not  Used 


CARD COLUMNS 
1 

4-11 
12 
13-30 
31-36 
37  42 
4349 
50-52 
53-55 
56-57 
58-59 
60-61 
62-63 
64-65 
66-69 
70-73 


Number  of  Tin 


APPENDIX  III.  TIED  AVERAGE  BOARD  SCORES  AT  PROMOTION  QUOTA  CUTOFF 


L*gand 


Quota  Cutoff 


13 
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»3  ABSTRACT 


For  the  purposes  of  field  testing  the  proposed  Weighted  Airman  Promotion  System,  the  Alaskan  Air  Command 
supplied  data  on  2.835  airmen  eligible  for  promotion  to  grades  H-4  through  E-7  in  the  FY  69-B  promotion  cycle. 
Data  included  scores  on  the  Specialty  Knowledge  Test  and  the  Promotion  Fitness  Examination;  points  credited  for 
time  in  grade,  time  in  service,  decorations,  and  Airman  Performance  Report  mean  overall  evaluation;  and  ID 
information.  Cases  for  which  test  scores  were  unavailable  were  eliminated;  the  final  sample  consisted  of  2,290 
airmen.  81  percent  of  the  original  eligibles. 

Average  promotion  board  scores  were  computed  by  dividing  each  airman’s  raw  board  score  by  the  number  of 
members  on  the  promotion  board  and  multiplying  the  quotient  by  10.  Weighted  factors  scores  were  computed  by 
adding  the  factor  scores.  Two  such  total  composite  scores  were  obtained:  a  weighted  factors  score  excluding  the 
average  board  score  and  a  weighted  factors  score  including  the  average  board  score.  The  analyses  concentrated  on  the 
comparisons  of  these  two  total  composite  scores.  The  hypothesis  tested  was  that  inclusion  of  the  board  score 
component  in  the  weighted  factors  composite  score  docs  not  have  an  effect  on  the  ranking  of  the  airmen.  Ranks  on 
the  composite  total  scores  with  and  without  the  board  score  were  analyzed  for  strength  of  relationship.  Average 
board  scores  were  analyzed  for  differences  in  scoring  across  Air  Force  Bases  and  by  different  size  board  panels. 

The  following  results  were  obtained:  (a)  There  were  significant  differences  in  average  board  score  means 
between  Elmendorf  and  Eiclson  AFBs  for  grade  E-3  personnel,  (b)  There  were  significant  differences  in  average 
board  score  means  between  3-inernbcr  and  5-mcmber  board  panels,  (c)  There  was  a  very  high,  near  perfect 
relationship  between  weighted  factors  composites  excluding  and  including  the  board  score  component,  (d)  Rankings 
of  individuals  on  the  weighted  factors  composite  were  essentially  unaffected  by  inclusion  of  the  promotion  board 
score. 

Since  inclusion  of  a  board  score  ^emponent  had  a  negligible  effect  on  the  ranking  by  composite  total  score, 
and  since  means  of  board  scores  differed  between  bases  and  between  boards  with  different  panel  sizes,  it  was 
concluded  that  including  a  board  score  factor  merely  inserted  a  non-visible  component  which  would  obscure 
explanations  for  non-promotion. 
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